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ABSTRACT 

A transition program at the American College of Life 
Underwriters which is moving into criterion-referenced measurement, 
but is also maintaining some of the desirable aspects of 
norm- referenced measurement,' is discussed. Two programs that have 
moved into the criterion-referenced measurement area are the 
Cassette/Review Program of the Zimmerman Adult Learning laboratory 
and the Mathematics of Life Insurance portion of ci?e of the C.L.U. 
courses. (Author/DB) 
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IMPLEMENTING A MIXED PROGRAM 
OF 

CRITERION AND NONCRITERION-REFERENCED MEASUREMENT* 



Harold F. Rahmlov: 

Zimmerman Adult Learning Laboratory 
American College of Life Underwriters 
Bryri Mawr, Pennsylvania 

In recent years there has been an upsurge of interest in criterion- 
referenced measurement. This movement, while basically healthy, is at the 
same time dangerous. Garvin pointed out, there are cases where criterion- 
referenced measurement is irrelevant because there are in fact no meaningful 
criteria (Garvin, 1971). If there are meaningful criteria against which to 
judge performance, however, norm-referenced measurement is inappropriate. 
Garvin maintains that the question is not whether we should use criterion- 
referenced measurement or norm- referenced measurement, but rather when each 
should be used. In a paper presented to this AERA forum in 1963 and later 
published in the American Psychologist , Glaser detailed the essential 
difference between criterion and norm-referenced measurement and recommended 
situations in which each might be used (Glaser, 1963) • 
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Making a rational decision to use criterion- referenced measurement 
is in theory easy. In practice, it may be very difficult. As Cox has 
pointed out, (Cox, 1971) most examples of criterion-referenced testing have 
been associated with individualized instructional programs. Moreover, most 
instances of criterion-referenced measurement are associated with new rather 
than ongoing programs. But what should be done about established programs? 
Since there are ongoing situations in which criterion-referenced measurement 
might be more appropriate than norm- referenced measurement, it would seem 
desirable to examine these situations, decide whether or not this is the 
appropriate method of measurement and if criterion- referenced measurement 
would be more appropriate, move toward this goa -. The concept of transition 
from a norm-referenced measurement program to one which may be a mixed norm- 
referenced and criterion-referenced measurement is generally not discussed 
in great detiil. It is, however, unrealistic to assume that a program which 
is norm- referenced can be, with a wave of a wand, transformed into a 
criterion- referenced program. Oftentimes, although it may be felt there are 
criteria which could be used, the problems associated with validating those 
criteria and developing the appropriate measures to assess student performance 
are horrendous. 

What I would like to describe for you today is a transition program 
at the American College of Life Underwriters which is moving into criterion- 
referenced measurement, while not abandoning some of the desirable aspects 

of norm-referenced measurement. 

Since 1927, the American College of Life Underwriters has been 

administering a program of examinations leading to the Chartered Life Under- 
witer Diploma (C.L.U.), (Stone, 1963). The program has been built upon 
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identified needs, but has not made explicit the objectives upon which 
the evaluation is based. Therefore, although the measurement suggested 
a criterion-referenced approach, it could not truly be called criterion- 
referenced. 

Recently, the College established the Zimmerman Adult Learning 
Laboratory for the purpose of improving the quality of adult education 
thro ugh basic research and through improved instructional programs (Lucas, 
1971, Rahmlow, 1971) . In order to accomplish this mission and to document 
improved learning results, it is necessary to begin with criteria against 
which to judge performance. This need is leading the College toward 
criterion-referenced measurement. This paper will describe two programs 
that have moved into the criterion-referenced area. It will also describe 
the larger picture of criterion documentation. 

CASSETTE /REVIEW PROGRAM 

A recent important project of the Zimmerman Adult Learning 
Laboratory has been to develop an audio Cassette/ Review Program which 
provides a review for the 10 course C.L.U. Diploma Program. Beginning 
with the general objectives for the C.L.U. Diploma Program courses, 
specific criteria against which candidate performance could be measured 
were developed. The initial task of this project was to ask Subject Matter 
Experts to generate questions which could be used to measure a candidate's 
comprehension of the subject matter. After these questions were determined 
other members of the learning systems team worked with the Subject Matter 
Experts to specify objectives encompassing a number of specific questions. 
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Through this interactive process a set of objectives, along with test 
items referenced to these objectives was developed. Within this frame- 
work, the review program was developed (Langdon, in press) . 

The evaluation of the effectiveness of the audio Cassette/Review 
Program was judged in the normal pre-test, post-test manner initially 
using a relatively small sample. The data was then used for improving 
the review program. 

However, the criteria against which the review program should 
ultimately be judged is the candidates* performance on the national C.L.U. 
Diploma examination. A study of this kind is underway. These national 
examinations, however, are based on general rather than specific objectives 
and cannot be judged to be criterion— referenced tests. So in this first 
example we see an instructional program having as its criteria for success 
a noncriterion-referenced test. 

MATHEMATICS OF LIFE INSURANCE 

Let us consider another instance where the development of instruc- 
tional programs has generated the need for developing criterion-referenced 
measurement. Currently, the C.L.U. Diploma Program is supported by group 
oriented instructional materials, but data indicates that approximately 50% 
of the College's candidates do not participate in formal group instruction. 
Therefore, it is logical to develop additional materials aimed at individual 
candidates rather than at groups. As a pilot project, individualized 
materials are being developed for the mathematics of life insurance portion 
of one of the C.L.U. courses. The first step in the development of these 
materials was a derivation of objectives for the course. At the same time 
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objectives w ere being developed, test items reflecting those objectives 
were being generated. The objectives and items then provide a basis 
for evaluating the effectiveness of the instructional materials* Again, 
the reason for developing the instructional materials is to help candidates 
pass the C.L.U* Diploma examinations* This once more poses a problem of 
criterion transfer. 

ACHIEVEMENT TESTING 

So far we have discussed two criterion-referenced instructional 
programs designed to assist candidates in developing competency to pass 
a noncriterion- referenced test* Since this noncriterion-referenced test 
is developed independently, it can be argued that the achievement test is 
itself the criterion against which candidate performance is being judged. 
This line of reasoning can be further supported when it is considered that 
the awarding of the C.L.U* Diploma is, in part, based upon the successful 

completion of the 10 examination series. 

Consequently, it can be argued that the C.L.U. examination is the 
criteria which the total instructional program should be judged. But is 
an ex ami nation an appropriate ultimate criteria? The College has felt 
the answer is no. Throughout its history, the American College has sought 
to establish a relationship between job performance and the C.L.U. deigna- 
tion. While the work necessary to establish a definite causal relation- 
ship has not been undertaken, periodic surveys indicate the strong 
possibility of such a relationship (Lucas, 1970) . 
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The reason for the existence of the C.L.U. Diploma Program is 
to educate persons in the insurance field so that they may serve the 
public more effectively. The real criterion, against which the exam- 
ination and therefore the total instructional program should be judged, 
is service to the public. This concept obviously poses some very 
stickey problems of criteria definition, but we are making some progress. 

SUMMARY 

The purpose of this paper was to summarize some of the activity 
taking place at the American College of Life Underwriters in an effort 
to introduce the concept of criterion-referenced measurement to a program 
that has not been criterion-referenced. As we have seen, this introduction 
of criterion-referenced measurement has moved the College’s measurement 
activities from a noncriterion-referenced mode to a mixed mode. One of 
the strong forces \diich influenced this move toward criterion-referenced 
measurement was a need to document progress in Improving instructional 
materials. In addition, there has been a search for a basis for a 
realistic criteria against which the total program could be evaluated. 
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